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NASA’s Orion spacecraft is designed to autonomously rendezvous and dock with many 
vehicles including the International Space Station. However, the crew is able to assume 
manual control of the vehicle’s attitude and flight path. In these instances, Orion must meet 
handling qualities requirements established by NASA. Two handling qualities assessments 
were conducted at the Johnson Space Center to evaluate preliminary designs of the vehicle 
using a six degree of freedom, high-fidelity guidance, navigation, and control simulation. The 
first assessed Orion’s handling qualities during the last 20 ft before docking, and included 
both steady and oscillatory motions of the docking target. The second focused on manual 
acquisition of the docking axis during the proximity operations phase and subsequent 
station-keeping. Cooper-Harper handling qualities ratings, workload ratings and comments 
were provided by 10 evaluation pilots for the docking study and 5 evaluation pilots for the 
proximity operations study. For the docking task, both cases received 90% Level 1 
(satisfactory) handling qualities ratings, exceeding NASA’s requirement. All ratings for the 
ProxOps task were Level 1. These evaluations indicate that Orion is on course to meet 
NASA’s handling quality requirements for ProxOps and docking. 


I. Introduction 

NASA’s Constellation Program had a goal to provide sufficient and consistent handling qualities for manual 
controllable operations across all new Constellation vehicles, which included the Orion spacecraft. The Orion 
spacecraft has been designed to rendezvous and dock with other Constellation vehicles in low earth orbit and 
beyond, including the International Space Station (ISS). Although Orion has autonomous docking capabilities, it is 
expected that final approach and docking operations with the ISS will initially be performed manually. 
Furthermore, NASA’s Orion Program includes a specific system requirement 1,2 that Orion must provide manual 
control for proximity operations and docking, while other human rating and integration requirements mandated that 
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handling qualities ratings required during manual control must be Level 1 during nominal operations (such as 
docking). 

As published in their seminal 1969 paper “The Use of Pilot Rating in the Evaluation of Aircraft Handling 
Qualities,” authors Cooper and Harper established the definition of handling qualities as “those qualities or 
characteristics of an aircraft that govern the ease and precision with which a pilot is able to perform the tasks 
required in support of an aircraft role.” 3 What the pilot feels (vehicle response), what the pilot sees (out- the -window 
& displays), and what the pilot touches (input devices) are handling qualities factors. 

Traditionally assessed using military-trained test pilots, handling qualities have been an important factor in 
spacecraft design and development from the earliest days of the space program. This legacy is demonstrated by 
NASA's use of test pilots as its piloting astronauts starting with Mercury and continuing with Gemini, Apollo, and 
Space Shuttle. Some of the initial work to support manual piloting of spacecraft addressed rendezvous and docking 
operations, 4 " 8 manual control modes, 9 ' 14 use of video for docking operations when direct view is not possible, 14 ’ 15 
visual aids to enable direct view, failed sensors, 9 ’ 11 " 13 ’ 15 flight control parameters, 16 bounding of attitude 
oscillation, 10 failed propulsion components such as jets or thrusters, 9 and the overall assessment of handling 
qualities for Gemini and Apollo. 9,12 ’ 14 Much of what was learned in these early programs pertaining to manual 
piloting was applied to the Space Shuttle Program. 7 Subsequently, the design of the Orion systems enabling manual 
piloting during proximity operations and docking was largely influenced by Shuttle-to-Mir and Shuttle-to-ISS 
development and operational experience. 17 " 18 

A series of evaluations were conducted at the Johnson Space Center to determine the handling qualities (HQ) of 
the Orion spacecraft during different docking and proximity operations conditions using the Cooper-Harper rating 
scale. This paper addresses the specifics of the handling qualities methodology, simulation environment, scenarios 
flown, data collection tools, and subject ratings and comments from these evaluations 


II. Methodology 

This evaluation was part of the Orion Spacecraft Handling Qualities Assessment. The objective of the 
assessment was to evaluate the handling qualities of the Orion spacecraft iteratively throughout the Orion design 
phase. This iterative assessment process (see Figure 1) is intended to be conducted early in the design process and 
repeated as necessary in order to identify and mitigate potential handling qualities problems so that engineering 
modifications can be made to assure requirement compliance during the verification phase of the Orion project. 
This paper discusses the current handling qualities results of the Rendezvous, Proximity Operations, and Docking 
(RPOD) portion of this assessment. The goal of the evaluation was to determine if the Guidance, Navigation, and 
Control (GN&C) design and vehicle configuration can meet spacecraft handling performance requirements (i.e. 
thruster configuration, sufficient control authority, etc.) set forth by NASA. 



Figure 1. Orion handling qualities evaluations testing program 
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The RPOD portion of the handling qualities assessment consists of two separate evaluations: one focusing on 
final approach for docking Orion to ISS (Docking evaluation) and the second focusing on acquisition of the docking 
axis (ProxOps evaluation). The Docking and ProxOps evaluations were performed during the summers of 2008 and 
2010 respectively. The planning and execution of these two evaluations were a collaborative NASA and Lockheed 
Martin (LM) endeavor which included multiple engineering disciplines from each. Key roles in the evaluation were 
performed by the Astronaut Office, Guidance, Navigation and Control (GN&C) Engineers, Human Factors 
Engineers, Simulation Engineers with consulting from Ames Research Center (ARC) and Langley Research Center 
(LaRC) Spacecraft Handling Qualities (SHaQ) group. 

Astronauts served in three different roles during the development and execution of these evaluations. Some 
astronauts were involved in the planning and development, some participated in the dry run activities and others 
were involved as evaluation pilots in the evaluation. Only astronauts who had not participated in the development or 
dry run activities served as evaluation pilot. For each of the two evaluations, two shuttle commanders with test 
piloting experience served as dry run evaluators. Ten astronauts evaluated the Docking scenario, while five assessed 
the ProxOps scenario. All evaluation pilots were experienced astronauts and the majority of them (11 of 15) were 
experienced military test pilots. All evaluation pilots were briefed on the evaluation, informed of their rights as 
subjects in the evaluation, and asked to sign informed consent forms. The protocol for these assessments was 
approved by NASA’s Committee for the Protection of Human Subjects (NASA’s internal Institutional Review 
Board (IRB) for research involving human subjects). 

Preceding the evaluations was a period of requirements review in which the mandate for manual control was 
assessed, as well as a period of task review in which we looked at roughly 100 potential manual control tasks during 
a mission of the Orion capsule to ISS and back. These 100 manual control tasks were then reduced to about 30 
based on the likelihood of them being needed or allowing for manual control within vehicle margins, and finally 
down to 1 1 key tasks that the team felt were the core manual control tasks that the program needed to evaluate and 
verify. The evaluation flow can be seen in Figure 2. 
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Figure 2. Evaluation flow 


Evaluation planning then began, with each evaluation broken into three distinct phases: evaluation 
development, dry runs, and formal evaluation. The evaluation development phase included discussion and 
assessment of what types of manual control scenarios to investigate, identification of appropriate performance 
metrics for each scenario, and design/development of simulations for candidate scenarios. Once the scenarios were 
simulated and the metrics derived, dry run assessments were used to evaluate the scenario simulations and 
evaluation procedures. Feedback from the dry run evaluators was used to improve subject briefing packages, 
scenario-specific piloting metrics, and evaluation time estimates. At the start of each formal evaluation session, the 
subject received an Orion Spacecraft Handling Qualities Assessment overview, a briefing on the 1969 Cooper- 
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Harper (CH) rating scale, and a scenario-specific briefing. Following the briefings, the subject was given unlimited 
familiarization time in the simulator to become comfortable with the controls, displays, flight tasks being evaluated, 
and to be exposed to the different cases within the scenario. 

Ratings were not collected during the familiarization runs and the subject was allowed to experiment with 
different flight techniques during each run. Once the subject indicated comfort with the scenario, the subject would 
perform a case-specific practice run for the particular case under evaluation. After completing the practice run, the 
subject would perform two data runs with an option for a third data run. If the subject was satisfied with his practice 
run performance, the subject could elect to use the practice run as the first data run. If the subject was dissatisfied 
with the practice run, an additional practice run was offered. CH ratings were collected after the completion of all 
data runs for each case. Once the subject’s CH ratings and comments were collected, the subject would start 
practice runs on the next case. Objective data (described later) were collected on every familiarization, practice, and 
data run, but only subjective data collected during the data runs will be discussed in this report. Subject comments 
were collected throughout the entire evaluation, including during familiarization runs. Cases and initial conditions 
were randomly assigned to each subject and every subject evaluated every case (but not every subject saw every 
initial condition). After the completion of all cases, the subject was debriefed on the results of the session. Before 
any scripted questions were asked, the subject was asked for observations and comments on the scenario. The pilot 
was then asked scripted questions from a questionnaire as well as unscripted follow-up questions from the test team. 


III. Simulation and Facilities 

The simulation used the Advanced NASA Technology Architecture for Exploration Studies (ANT ARES), 19-20 
which is the NASA high-fidelity, six degrees-of- freedom simulation used for Orion GNC algorithm development 
and analysis. It has been designed to support a wide variety of requirements assessments, design trades, and 
operator-in-the-loop evaluations for the Orion Crew Module (CM). ANTARES is based on a Common Model 
Library architecture and the Trick simulation tool kit which leverages off the significant simulation development 
conducted by several organizations across JSC, including the Aeroscience and Flight Mechanics Division, the 
Automation, Robotics and Simulation Division and the Mission Operations Directorate. In 2008, it won the JSC 
Exceptional Software Award, the NASA Exceptional Technology Achievement Medal, and the NASA Agency 
Software of the Year. It is used for Orion GN&C algorithm development and analysis. Although primarily a batch 
simulation, ANTARES can be used for real-time pilot-in-the-loop evaluation with a Translational Hand Controller 
(THC), Rotational Hand Controller (RHC), notional displays for piloting cues, flight control system configuration 
(pilot selected translational authority: coarse/fme), and real-time graphics for simulated out-the- window and docking 
camera views. ANTARES was used for development of the manual control algorithm prototypes used in the 
selected scenarios, determining the Cooper-Harper criteria, and specific cases to be used for the evaluations. 
Current sets of mass properties, initial vehicle state, aerodynamics, atmospheric, and navigation sensor models were 
included into the ANTARES simulation. Nominal and dispersion sets for all important parameters were 
implemented. 

NASA’s Reconfigurable Operational Cockpit (ROC) facility was used for both assessments to provide the 
Orion cockpit environment. The three Orion display units in the ROC are capable of showing Orion displays, 
overlays, and docking camera views. The ROC (see Figure 3) also provides out-the -window views for situational 
awareness using image projection. 
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Figure 3. The ROC facility. The left shows the actual setup (illuminated by the camera flash) and the right 
shows the type of out-the-window scene generation capability that was used. 

Cooper-Harper Rating Scale 

The assessment of handling qualities was performed using the 1969 Cooper-Harper rating scale. 3 The Cooper- 
Harper rating scale is the most commonly used handling qualities assessment tool and is required by NASA for 
spacecraft human rating. 16 The Cooper-Harper decision tree (Figure 4) begins with an assessment of the vehicle’s 
“Adequacy for Selected Tasks or Required Operation” in which the evaluation pilot decides if the performance 
achieved in a piloting run was desired, adequate or uncontrollable (these are adjectives associated with performance 
and used as anchors at various locations within the scale). Performance is defined as the precision of control with 
respect to flight vehicle movement that a pilot is able to achieve in performing a task. Adequate performance means 
that the end result obtained was sufficient for successful completion of the required operation or task. Desired 
performance means that the end result obtained exceeded the level of performance needed for successful completion 
of the required operation or task. Since these terms (desired/adequate performance) are involved in the core 
decision logic of the scale, the CH performance metrics associated with their definition are key drivers of the rating 
process. Following the initial determination of operational adequacy, the evaluation pilot subject then proceeded to 
the right of the scale into the defined categories of Level 1, 2 or 3. Next the evaluation pilot reviewed the flight 
vehicle characteristics, the demands on the pilot, and determined the final rating. The Cooper-Harper protocol 
allows for half-ratings within a level (1.5, 2.5, 4.5, 5.5, 7.5, 8.5), but does not allow half-ratings across levels (3.5, 
6.5, 9.5). 

In handling qualities experiments, evaluation pilots are typically asked to make a composite assessment of the 
overall performance across all formal evaluation runs for a test configuration. This assessment takes into account 
not just the quantitative evaluation of the performance at the end point (e.g., hard dock) but also a qualitative 
evaluation of the manner in which the vehicle gets to the end point. This overall assessment of desired, adequate, or 
inadequate performance is utilized for traversing the decision tree in the Cooper-Harper chart. 3 Pilots use the 
Cooper-Harper scale to assign handling qualities ratings from 1 (best) to 10 (worst) based on their assessment of 
task performance and required compensation. It is an ordinal, non-linear scale, such that the difference between 
ratings 1 and 2 is not the same as the difference between ratings 3 and 4, thus the data collected using the scale must 
be assessed using non-parametric methods which lack distributional assumptions of normality or linearity. 
Additionally, the ratings are categorized into levels, such that ratings of 1, 2, and 3 on the Cooper-Harper scale 
correspond to Level 1 handling qualities, which are a general requirement for normal operations of flight vehicles. 
Desired performance is necessary, but not sufficient, for Level 1 ratings. Ratings of 4, 5, and 6 correspond to Level 
2 which may be acceptable for some off-nominal conditions, while ratings of 7, 8, and 9 correspond to Level 3 
which is acceptable only for transition to a safe mode after a major failure or disturbance. 
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Handling Qualities Rating Scale 
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require 
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Major deficiencies 
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required for control 
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Intense pilot compensation is required to 
retain control 


Control will be lost during some portion of 
required operation 


Pilot decisions 


* Definition of required operation involves designation of flight phase 
and/or subphases with accompanying conditions. 


Figure 4. Cooper-Harper Rating Scale reproduced from NASA TN D-5153 3 


Workload 

The NASA Task Load Index (TLX) 21 and Bedford Pilot Workload Rating Scales 22 " 23 were also used for the 
Docking and ProxOps scenarios respectively. The reason the workload measures were not consistent between the 
evaluations is because the NASA requirements governing workload changed between the two evaluations. During 
the earlier Docking Evaluation, NASA required TLX, but later, prior to the ProxOps evaluation, the NASA 
requirements for workload changed to the Bedford Workload Scale. Workload was not of the main objective of 
these evaluations, but is an important measure that can contribute to the handling qualities ratings and thus is useful 
to understand. 

The NASA Task Load Index is a weighted average of six workload components: physical demand, mental 
demand, temporal demand, performance, effort, and frustration. 21 The relative weighting of these six components 
for the flying task was determined by a pilot questionnaire at the end of the experiment. Each component is 
assigned a rating between 0 and 100. Unlike the Cooper-Harper or Bedford scales, there are no descriptors 
associated with TLX component ratings; there is only a low/high anchoring on the ends of the scale. A rating of 
zero corresponds to low, and a rating of 1 00 corresponds to high. The rating for performance is subtracted from 1 00 
prior to computing the weighted average. 

The Bedford Pilot Workload Rating Scale (Figure 5) was utilized to assess the workload associated with the 
ProxOps manual control task. Workload is an important measure that can contribute to the handling qualities ratings 
and thus is useful to understand. The Bedford Rating Scale was administered exactly like the Cooper-Harper Scale. 
Evaluation pilots were asked to start at the bottom and walk through each logic gate. Unlike CH, Bedford is not 
broken up into levels and half ratings between decision gates are acceptable (e.g. ratings of 3.5 or 6.5). 22-23 
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Figure 5. Bedford Pilot Workload Rating Scale 
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Additional Data 

Several objective simulation variables were monitored and collected to facilitate interpretation of subjective 
handling qualities ratings and comments. The ANTARES data relevant to the simulation itself (e.g. vehicle state 
parameters, flight path information, etc.) were electronically recorded. The collection of these variables was 
programmed into the simulation by ANTARES programmers and GN&C specialists. The data collected included 
objective performance criteria used in evaluation for determining the adequacy of performance for Cooper-Harper 
rating purposes. Relevant data collected includes: 

• Timing information (duration of runs, time stamps for hand controller inputs) 

• Hand controller inputs (type, direction, magnitude, frequency) 

• Control system mode (pilot selected translational authority: coarse/fme) 

• T raj ectory and performance 

o Vehicle relative trajectory (position and velocity) 
o Vehicle orientation (attitude and rotational rates) 
o Vehicle control errors 
o Propellant used/remaining 

Additional subjective data was collected by the test conductor. Evaluation pilots’ verbal comments were 
digitally recorded throughout the evaluation and were also collected by test team note takers. The pre-brief was the 
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only portion of the evaluation where evaluation pilots’ comments were not recorded. While video of the evaluation 
was transmitted to test team members monitoring the evaluation from outside of the ROC, the video was not 
recorded. 

Assumptions and Constraints 

The primary outcome of this evaluation was an assessment of Cooper-Harper ratings associated with Orion’s 
handling qualities, given the evaluation’s constraints and assumptions. The computer simulation models, derived 
from ANTARES version 9.2.0, were high fidelity and accurate with regards to the physics of Orion under various 
flight conditions and use of actual Orion GN&C flight control algorithms. For these manually piloted scenarios, 
relative-navigation’s main purpose was to drive trajectory data on the crew displays. However, due to navigation 
model certification schedules, a decision was made to use a perfect relative navigation model instead of the Orion 
baseline relative navigation flight software. The perfect relative navigation model simply replaces internal relative 
navigation state estimates (position and velocity) with the true state generated from the simulation equations of 
motion, thus producing a perfect navigation output (no noise or bias errors). To be certain that this decision would 
not impact the study results, a visual comparison of the display output of both of these navigation models (perfect 
and Orion baseline) was conducted. The comparison showed no noticeable difference to engineer observers so the 
fidelity of the model was not expected to influence the evaluation results. Also, the GN&C team’s analysis of 
propellant slosh effects on flight control performance concluded that propellant slosh had an insignificant affect on 
flight control; therefore, a propellant slosh model was not included for this study. 

Space suits, gloves, and helmets were not employed in these evaluations. Evaluation pilots wore typical office 
clothing. Nominally, Orion flight crew would be fully suited for all scenarios evaluated, but suits were not 
incorporated into the evaluation primarily due to suit availability and resources. Future HQ evaluations will 
consider using suited evaluation pilots as it is anticipated this will be required for final vehicle verification. 

Scenarios 

The initial Orion HQ evaluation (Docking Scenario) examined docking Orion to the ISS. This scenario 
demonstrated the translational motion handling qualities of Orion. During the Docking scenario, two different cases 
were evaluated. The first case was a docking approach to a fixed-attitude ISS (Stable ISS), with Orion initializing 
with relative position dispersions (a 4 degree offset, which is not nominal, but added some stress to the piloting task 
to expose any underlying handling qualities issues) and a closing rate of approximately 0.1 ft/sec. Previous shuttle 
docking operations have shown the ISS motion to be imperceptible by the shuttle astronauts. For this reason, the 
ISS was considered a stable platform during the first docking scenario. The second docking case was identical to 
the first, except that the attitude motion of the ISS was modeled to simulate a highly perturbed case (± 1 degree 
deadband per axis and ± 0.01 deg/sec rate deadband per axis) with the ISS attitude-hold control function engaged 
(Perturbed ISS). For both cases, evaluation pilots started each run on final approach at a docking port-to-port range 
of 20 ft. Evaluation pilots used the THC in pulse mode with cues from the docking camera image, window views, 
and range and range rate data presented on the Orion display units. As in the actual Orion design, the attitude of the 
vehicle was held by the automated flight control system at ± 0.5 degree deadband per axis. Several error sources 
were modeled including Reaction Control System (RCS) jet angular and position misalignment, RCS thrust 
magnitude uncertainty, RCS jet force direction uncertainty due to self plume impingement, and Orion center-of- 
mass uncertainty. Both Docking cases used the same objective performance criteria, listed in Table 1. Table 2 
provides the translational pulse size settings for this scenario. Evaluation pilots were allowed to switch between 
these settings to achieve their desired pulse size. The Docking cases and initial conditions were presented to 
evaluation pilots in a random order. 

Evaluation pilots used two displays (Figure 6 and 7) to perform the docking procedure: a view from the docking 
camera with cross-hairs reticle overlay pointing outward along the centerline of the Orion docking port (Orb Cam) 
and a Primary Flight Display (Orb PFD). The centerline camera gave the evaluation pilots a visual perspective of 
the docking port, providing lateral translational cues, while the PFD provided the evaluation pilots with a digital 
indication of its current relative position and velocity. 
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T able 1. Cooper-Harper performance criteria at docking port contact for Docking scenar io 


Criteria 

Desired 

Adequate 

Lateral Offset 

1 inch from center 

2 inches from center 

Axial Closure Rate 

0.1 ft/s ± 0.01 ft/s 

0.1 ft/s ± 0.02 ft/s 


Table 2. Translational pulse size settings for the Docking scenario 


Translational Pulse Sizes 

Stable ISS 

Perturbed ISS 

Coarse 

0.01 ft/s 

0.03 ft/s 

Fine 

0.005 ft/s 

0.01 ft/s 




DOCKING 

R ET OOO/OO : 00 : 5 5 

RNG = 
RDOT = 

14.38 ft 
-O.IO ft/s 

R : 
P: 
Y : 

CUR CMD ERROR RATE 

0.08 -0.00 0.08 0.002 

0.21 -0.00 0.21 0.004 

0.31 0.00 0.31 -0.004 


DP-DP POS DP-DP VEL 

X : 

14.36 ft -0. 102 ft/s 

Y : 

0.46 ft -0.021 ft/s 

Z : 

0.65 ft -0 . 017 ft/s 

FUEL= 

1.03 lbs 

10 FRAME: 

11 UNITS: 

: DP REL 
: en gl i sh 


Flight | Ascent | Orbit J Entry | Test 


Figure 6. Engineering display of the docking 


Figure 7. Engineering PFD showing docking 


camera view used for docking 


rates and offsets 


The second evaluation (ProxOps scenario) evaluated the handling qualities of Orion while acquiring the 
docking axis. This scenario demonstrates the coarse translational control authority by arresting the lateral motion 
during the Acquisition of Docking Axis (ADA) maneuver on the docking axis (see Figure 8). Evaluation pilots used 
the THC in a pulse mode to acquire the docking axis, and then arrest closure rate and perform station-keeping. As 
with both the Docking scenario, the Orion’s attitude was automatically controlled by the flight control system. The 
target (ISS) was in a Torque Equilibrium Attitude of zero degrees of yaw and roll and negative seven degrees of 
pitch with respect to the Local Vertical Local Horizontal frame. The ProxOps scenario modeled several error 
sources including RCS jet angular misalignment, RCS jet position, RCS thrust uncertainty, RCS jet force direction 
uncertainty due to self plume impingement, Orion center of mass uncertainty, initial relative state dispersions (Orion 
with respect to ISS position, velocity, and attitude), and ISS attitude control deadbanding (within the range used for 
the Perturbed ISS case from the Docking Scenario). 
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ADA (Acquisition of Docking Axis) 



Figure 8. ProxOps scenario flight profile 


Evaluation pilots were provided with three displays to perform the ProxOps task; a view from the docking 
camera pointing outward along the centerline of the Orion docking port (Orb Cam) similar to Figure 3, an orbit 
relative trajectory (Orb Traj) display, and an updated orbit Primary Flight Display (Orb PFD). The centerline 
camera gave the evaluation pilots a visual perspective of the docking axis acquisition maneuver, while the Orb Traj 
provided the evaluation pilots with a graphical representation of Orion’s past, current, and future trajectory relative 
to the ISS, as well as a digital indication of its current relative position and velocity. The Orb PFD provided the 
subject with a countdown clock that indicated when Orion would be in docking attitude (In Att, MM:SS), but many 
of the other features present on the display were not operational. 

The ProxOps scenario only contained one case with several different initial conditions (ICs). The scenario 
would initialize about two minutes prior to Orion arrival on the docking axis at a range of 300 ft with a closing rate 
of approximately 0.50 feet per second. During the acquisition of the docking axis, the Orion flight control system 
would automatically maneuver Orion’s initial attitude to the required docking axis attitude with a pitch rate of 0.13 
degrees per second. Depending on the initial Orion attitude, Orion would arrive, positionally, on the docking axis 
before achieving the desired docking axis attitude (early case), or arrive at the docking axis after completing the 
attitude maneuver (late case). Several of the initial conditions included out-of-plane dispersions. In all, the ProxOps 
scenario contained five different ICs that were randomly presented to the evaluation pilots. The available ICs were: 

• Arrive on time without an out of plane offset (nominal) 

• Arrive Early with +y out-of-plane 

• Arrive Early with -y out-of-plane 

• Arrive late with +y out-of-plane 

• Arrive late with -y out-of-plane 
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Table 3 provides the translational pulse size settings for the ProxOps scenario. Evaluation pilots were allowed 
to switch between these settings to achieve their desired pulse size. Piloting techniques were not provided to the 
evaluation pilots for the ProxOps scenario. Evaluation pilots developed their own piloting techniques during their 
familiarization runs. Evaluation pilots were asked to acquire the docking axis and then perform station keeping 
(arrest closure rate). The Cooper-Harper performance criteria for this scenario are listed in Table 4. The desired and 
adequate criteria for position were developed by using a 2 degree and 3 degree corridor at a range of 250 feet 
respectively. It should be noted that these corridors are not requirements based or operationally based, but were 
chosen for this evaluation based on engineering judgment from NASA GN&C specialists. The criteria for relative 
velocity were selected to provide a reasonably small value that effectively results in station-keeping. Evaluation 
pilots could use the digital display of their position and velocities on the Orb Traj display to determine if they were 
meeting the performance criteria. 

Table 3. Translational pulse size settings for the ProxOps scenario 


Translational Pulse Sizes 

ProxOps 

Coarse 

0.05 ft/s 

Fine 

0.01 ft/s 


Table 4. ProxOps Cooper-Harper performance criteria 




Docking Port to Docking Port 


Position (ft) 

Velocity (ft/s) 


X 

- 


Xd 

<0.03 

Desired 

Y 

<9 


Yd 

<0.03 


Z 

<9 


Zd 

<0.03 







X 

- 


Xd 

<0.05 

Adequate 

Y 

<13 


Yd 

<0.05 


Z 

<13 


Zd 

<0.05 


IV. Results 

Appropriate statistical methods for analyzing and displaying Cooper-Harper ratings are constrained by the 
ordinal nature of the rating scales. As there is not a linear or other parametric relationship implied across the rating 
spectrum, only nonparametric ordinal methodologies are acceptable for analysis of Cooper-Harper ratings data. For 
this reason the data analyses have focused upon fundamental graphical and numerical assessment techniques such as 
frequency scatter plots. Likewise, due to the possibly non-linear of latent traits assessed via use of the TLX and 
Bedford scales, workload data was examined using graphical techniques such as multi-axis plots. 

Handling qualities ratings for the Docking and ProxOps scenarios are presented in Figure 9 where the size of 
the bubble for a given rating value indicates the number of evaluation pilots who assigned that rating, while the star 
symbol indicates the median rating for that category. For the two Docking cases, nine of ten evaluation pilots gave 
each case a Level 1 rating (rating of 1, 2, or 3) and one evaluation pilot gave each case a Level 2 rating. 
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Figure 9. Handling qualities ratings for ProxOps and Docking tasks 


Every run for the docking scenario had ‘desired’ performance for the CH metrics listed in Table 1. The 
evaluation pilots used their individual performance metric results to help them determine their Cooper-Harper 
Ratings. Subject comments for the stable ISS docking scenario were fairly consistent, with evaluation pilots noting 
very low cross-coupling of Translational Hand Controler command inputs into other vehicle attitude or translation 
axes. Many evaluation pilots indicated that the translational pulse sizes were appropriate to the task, though a few 
desired finer settings. For the Perturbed ISS case, several evaluation pilots noted that the translational pulse size 
settings were too aggressive and would have preferred a smaller pulse size. Due to the ISS motion and the greater 
number of inputs required to dock, several pilots noted some attitude dead-banding or mild coupling in this case. 
Evaluation pilots noted this task as more challenging than the stable ISS case, but still very achievable. Nonetheless, 
median values for both cases were within Cooper Harper Level 1, each with only a single rating (out of 10) in the 
Level 2 category. As can be seen in Figure 10, the Perturbed ISS case had a higher workload than the Stable ISS 
case, but overall, the workload for both cases was very low. 
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Figure 10. Task Load Index component ratings for docking task 


The ProxOps scenario results showed a solid Level 1 rating with every evaluation pilot meeting the desired 
Cooper-Harper criteria for every run. Evaluation pilots commented on the simplicity of performing this task. No 
handling qualities issues were uncovered during the evaluation. Two comments indicated that a narrower camera 
field of view (FOV) would be useful to monitor the motion of the target vehicle and to help determine the direction 
for nulling Orion’s translational rates. Evaluation pilots were only able to use a large FOV, but desired a smaller 
FOV. This capability will be available on Orion, but was not available during the evaluation. Several evaluation 
pilots requested a dynamic overlay, an Orion capability not available in this evaluation, showing Orion’s relative 
navigation estimates of the target on the centerline camera view. This would allow the subject to verify relative 
navigation system’s solutions were valid. Two evaluation pilots commented that the displays should include THC 
translation direction information. Several evaluation pilots (more than the two that commented) actuated the THC in 
a direction opposite from their desired vector. This polarity issue was most common in the y-axis, but would 
occasionally manifest in the z-axis. Evaluation pilots who commented on this polarity issue stated that their 
workload was increased because they had to think through which direction to actuate the THC to receive their 
desired rate. As can be seen in Figure 11, workload was extremely low for this task. 

Several comments were made that are already incorporated into the flight display prototypes, but were not 
available for this evaluation. Evaluation pilots requested an attitude deadband box on the camera view and that the 
time to attitude timer on the PFD be moved or repeated on the Traj display. 
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Figure 11. Bedford workload ratings for ProxOps task 
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V. Discussion 

The Docking scenario was assessed using both a ‘Stable ISS’ and a ‘Perturbed ISS case. The stable ISS case 
was included since observed station motion is very minimal, and this case was the most similar to conditions under 
which the astronauts would actually dock Orion to the ISS. The Perturbed ISS case was based upon the ISS 
specifications for stability (which date back to a time when the station had fewer modules and much less mass than 
it does today) which allow a certain degree of motion up to ± 1 degree of deadband per ISS axis and a ± 0.01 deg/sec 
rate deadband per ISS axis. 

The docking simulation also included several control-related error sources but did not include fuel slosh. Prior 
to the ProxOps evaluation, the GN&C team analyzed propellant slosh effects on flight control performance and 
concluded that propellant slosh had an insignificant affect on flight control. The stable ISS case findings included 9 
out of 10 ratings equal or better to a Handling Quality Rating (HQR) of 2, with one rating of 4. The evaluation 
pilots indicated that there was very low cross-coupling between translation and rotation and that Orion’s responses 
were very predictable, making docking a low workload task. 

The Perturbed ISS case was also rated well, with 9 out 10 ratings in the Level 1 category (HQRs of 3 or better) 
with one rating of 5. In general evaluation pilots rated this scenario one HQR higher (worse) than the stable ISS 
case, mainly due either to the larger translational pulse sizes (reported as too aggressive) and the ISS attitude 
deadbanding (not an Orion handling issue). The Perturbed ISS case required more THC inputs and a higher 
workload than the stable ISS case. Nevertheless, the Perturbed ISS case was reported as being a readily 
accomplished task. 

One evaluation pilot was responsible for both Level 2 ratings for the Docking scenario. The evaluation pilot’s 
comments indicated that better deadband control would lead to a rating of 2 (Level 1) for the Stable ISS case. 
During the Perturbed ISS case, the evaluation pilot saw the ISS change directions (ISS attitude jets fired to correct 
the ISS attitude) moments before contact causing the pilot to make a last minute translational correction. While this 
was not an issue with Orion handling qualities (Orion was able to make the pilot’s last second correction 
satisfactorily), the correction and increased workload impacted the CH rating. 
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In summary, the Docking scenario was generally rated as Level 1 (HQRs of 1-3), for which subject comments 
indicated that the docking cases were readily achievable. Additional options for pulse sizes and the inclusion of 
other docking display indicators would make the tasks even more pilotable. These pilot enhancements are available 
in the actual Orion flight software, but were not available to the evaluation pilots during this evaluation. 

The Docking evaluation demonstrated the fine translational control of Orion which is needed to complete 
docking operations with ISS and the ProxOps evaluation demonstrated the coarse translational control of Orion to 
arrest the relative motion during acquisition of the docking axis maneuver. No issues with the Service Module RCS 
translational authority were identified during the ProxOps scenario by evaluation pilots. The ProxOps scenario was 
rated as a Level 1 task by all evaluation pilots. Most of the subject comments were against the display format 
prototypes and not the handling of Orion. Probably the most significant comment was to better identify the polarity 
response of THC inputs, primarily in the y-axis. Several evaluation pilots commented that their workload was 
increased because they had to contemplate what the vehicle’s response polarity would be to an input in a given 
direction. Many of the other comments are being incorporated into the next iteration of flight displays. Some of 
these include an attitude deadband box on the centerline camera view, adding a time to desired attitude to the Orbit 
Traj, adding a 10 degree field of view zoom to the centerline camera, and a dynamic overlay showing Orion’s 
relative navigation’s estimate of the target vehicle’s location on the centerline camera view. 

Overall, the evaluation pilots of this study (all of which were trained NASA astronauts) indicated very minimal 
cross-coupling between translation and rotation and responses were very predictable, making ProxOps and docking 
low workload tasks. The tasks evaluated were generally rated as Level 1 indicating that the ProxOps and Docking 
scenarios were readily achievable with the fine and coarse translational authority provided by Orion. Based upon 
the reported Cooper-Harper Ratings, subject comments, and additional HQ assessments, Orion’s vehicle 
configuration appears to be on the path to achieving the HQ ratings desired for a human rated spacecraft. 


VI. Conclusion 

The goal of the evaluations presented here was to investigate the handling qualities evidenced by NASA’s 
Orion Crew Exploration Vehicle during a variety of rendezvous, proximity, and docking maneuvers. Both the 
ProxOps scenario as well as the actual docking scenarios demonstrated median Level 1 handling qualities, with all 
runs resulting in desired performance. Results were additionally associated with low evaluation pilot workload 
ratings and favorable subjective commentary. Nonetheless, room for improvement was still noted with requested 
addition of refined translational pulse control modes, and the addition/modification of visual aids represented on the 
docking displays. These findings have direct relevance to NASA’s continued development of the Orion spacecraft 
as its future deep space mission vehicle, recently renamed the “Multipurpose Crew Vehicle” (MPCV). In its future 
potential missions to near earth objects or planets such as Mars, Orion may be challenged with additional docking 
tasks such as rendezvous and docking/undocking with landing vehicles, robotic drones, or long duration spacecraft 
support systems (e.g. mobile habitat modules). As Orion’s mission changes, future handling qualities evaluations 
will build off of the current study to ensure that Orion maintains its excellent docking handling qualities allowing 
this most critical of in-space activities to be conducted safely and reliably, ensuring the success of future missions 
beyond Earth’s orbit. 
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